(b) 


August 2015 

Applied Research Methods 




Development and examination 
of an alternative school 
performance index in 
South Carolina 


Sharon Koon 
Yaacov Petscher 
John Hughes 

Florida Center for Reading Research 
at the Florida State University 


Key findings 

The measures that make up each of the three 
school performance indices currently used in 
South Carolina to rate schools can be consolidated 
using a bi-factor model to create an overall, reliable 
alternative index of school performance. 

nder the alternative index of school performance, 
about 3 percent of elementary schools, 2 percent 
of middle schools, and 3 percent of high schools 
performed better than expected (beat the odds), 
given the schools’ demographic characteristics. 
Most of the school demographic profiles identified 
by the study contained at least one school that 
was beating the odds. 


ries 


NATIONAL CENTER for 
EDUCATION EVALUATION 
and REGIONAL ASSISTANCE 


Institute of Education Science 
U.S. Department of Education 




U.S. Department of Education 

Arne Duncan, Secretary 

Institute of Education Sciences 

Ruth Neild, Deputy Director for Policy and Research 
Delegated Duties of the Director 

National Center for Education Evaluation and Regional Assistance 

Joy Lesnick, Acting Commissioner 
Amy Johnson, Action Editor 
Sandra Garcia, Project Officer 

REL 2015-097 

The National Center for Education Evaluation and Regional Assistance (NCEE) conducts 
unbiased large-scale evaluations of education programs and practices supported by federal 
funds; provides research-based technical assistance to educators and policymakers; and 
supports the synthesis and the widespread dissemination of the results of research and 
evaluation throughout the United States. 

August 2015 

This report was prepared for the Institute of Education Sciences (IES) under Contract 
ED-IES-12'C'OOll by Regional Educational Laboratory Southeast administered by the 
Florida Center for Reading Research, Florida State University. The content of the publi- 
cation does not necessarily reflect the views or policies of IES or the U.S. Department of 
Education nor does mention of trade names, commercial products, or organizations imply 
endorsement by the U.S. Government. 

This REL report is in the public domain. While permission to reprint this publication is 
not necessary, it should be cited as: 

Koon, S., Petscher, Y., Hughes, J. (2015). Development and examination of an alternative 
school performance index in South Carolina (REL 2015-097). Washington, DC: U.S. Depart- 
ment of Education, Institute of Education Sciences, National Center for Education Evalua- 
tion and Regional Assistance, Regional Educational Laboratory Southeast. Retrieved from 
http://ies.ed.gov/ncee/edlabs. 

This report is available on the Regional Educational Laboratory website at http://ies.ed.gov/ 
ncee/edlabs. 


Summary 


Administrators from the South Carolina Department of Education partnered with Region- 
al Educational Laboratory Southeast to test the extent to which the state’s measures of 
school performance could be consolidated into an overall, reliable index that can be used 
to rate schools under the state’s Elementary and Secondary Education Act waiver. In addi- 
tion, the department was interested in using the new index to identify schools that are 
high and low performing after school demographic characteristics are controlled for. 

At the time of this report’s publication, South Carolina rated school performance using 
three indices. The indices result in very different performance rankings of schools. While 
schools generally see consistent results between two of the index scores, scores on the third 
index are often unrelated or inversely related to the other two indices. Although the con- 
ceptual distinctions can be explained, communicating results to educators is challenging 
when ranking of schools is inconsistent across the indices. Empirical evidence of these 
inconsistencies can be seen in the correlations between index scores. 

This study used data from the South Carolina Department of Education on public ele- 
mentary schools (grades 3-5), middle schools (grades 6-8), and high schools (grades 9-12) 
for 2012/13 to determine whether the measures that make up each of the three indices of 
school performance in South Carolina can be used to create an overall, reliable alterna- 
tive index of school performance. The alternative index was then used to identify which 
schools’ performance scores on the alternative index are better than expected (that is, 
which schools are beating the odds) after school demographic characteristics are controlled 
for. The study also sought to identify distinct school demographic profiles among South 
Carolina schools and to ascertain which profiles include schools that are beating the odds. 

The study found that the measures that make up the three indices currently used in South 
Carolina to rate schools can be combined into an overall, reliable alternative index of 
school performance using a bi-factor model. The alternative school performance index 
identified approximately 3 percent of elementary schools, 2 percent of middle schools, and 
3 percent of high schools as statistically exceeding their expected performance after the 
schools’ demographic characteristics were accounted for. 

The study also found five distinct demographic profiles at the elementary school level, four 
at the middle school level, and three at the high school level. Most of the profiles included 
at least one school that was beating the odds. 
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Why this study? 


Administrators from the South Carolina Department of Education partnered with Region- 
al Educational Laboratory Southeast to test the extent to which the state’s three indices of 
school performance could be consolidated into an overall, reliable alternative index that 
can be used under the state’s Elementary and Secondary Education Act (ESEA) waiver. 
The waiver allows the state to use one index to meet state and federal school accountabil- 
ity requirements. The department has noted that school performance ratings under the 
current indices result in very different rankings of schools. The state’s goal is to leverage 
the existing indices to develop an overall, reliable index. In addition, the department was 
interested in using the alternative index to identify high-performing schools after school 
demographic characteristics are accounted for (that is, schools that are “beating the odds”). 

At the time of this report’s publication, South Carolina rated school performance using 
three indices: 

• Absolute performance index. At the elementary school (grades 3-5) and middle 
school (grades 6-8) levels absolute performance is measured by the number of stu- 
dents who meet a given performance level on the state assessment, the Palmetto 
Assessment of State Standards, which covers English language arts, math, science, 
and social science. At the high school (grades 9-12) level absolute performance is 
measured by a combination of annual student performance on statewide end-of- 
course exams and other exit exams, as well as by four-year and five-year graduation 
rates. 

• ESEA performance index. At all school levels ESEA performance is based on the 
percentage of student subgroups that meet the annual proficiency target on certain 
measures, such as statewide assessments in English language arts, math, science, and 
social science and graduation rates. For a given measure there are up to 10 demo- 
graphic subgroups (male students, female students, White students, Black students, 
Asian/Pacific Islander students, Hispanic students, American Indian/Alaskan stu- 
dents, disabled students, limited English proficient students, and students eligible 
for free or reduced-price lunch 1 ) in addition to the group of all students. Typically 
one or more subgroups are excluded because of small numbers of students. 

• Qrowth performance index. At the elementary and middle school levels growth 
performance is determined using a value table developed by the South Carolina 
Department of Education. The measure generates a student growth score based on 
a student’s prior year and current year scores on each of four statewide assessments 
(reading, math, science, and social science). At the high school level growth per- 
formance is based on the difference between the school’s current year and prior 
year’s absolute performance. As such, growth performance at the high school level 
is a cross-cohort comparison instead of a within-cohort comparison, as at the ele- 
mentary and middle school levels. 
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Table 1 describes the measures that make up each index by school level, and appendix A 
provides details on how the indices are calculated. All the indices are derived using a point 
system. 

Using three indices to evaluate school performance can be confusing to educators and 
other stakeholders. Although the conceptual distinctions can be explained, communicat- 
ing results to educators is challenging when ranking of schools is inconsistent across the 
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Table 1. Description of South Carolina school performance indices, 2012/13 

Index 

Elementary and middle schools 
(grades 3-8) 

High schools 
(grades 9-12) 

Absolute 

performance 

index 

Based on the number of students who 
meet a given performance level on state 
assessments. Performance by subject 
area is used to create a subject area index 
score, which is then used to create the 
overall absolute performance index score. 
The subject area index scores are writing 
and English language arts combined, math, 
science, and social science. 

Based on the percentage of students who 
pass state assessments and graduate 
on time or in five years. The individual 
measures are the longitudinal (spring) High 
School Assessment Program passage rate, 
the first attempt High School Assessment 
Program passage rate, the end-of-course 
exam overall passage rate, the four-year 
cohort graduation rate, and the five-year 
graduation rate. 

Elementary 
and Secondary 
Education Act 
performance 
index 

Based on the percentage of subgroups that 
meet the target for a state assessment 
and the percentage tested. The individual 
measures are English language arts 
weighted points (points earned multiplied 
by the specified weight), math weighted 
points, science weighted points, social 
science weighted points, English language 
arts percentage tested weighted points, 3 
and math percentage tested weighted 
points. 

Based on the percentage of subgroups that 
meet the target for a state assessment, 
the percentage tested, and the graduation 
rate. The individual measures are English 
language arts weighted points, math 
weighted points, science weighted points, 
social science weighted points, graduation 
rate weighted points, b English language 
arts percentage tested weighted points, 3 
and math percentage tested weighted 
points. 

Growth 

performance 

index 

Based on the amount of student growth 
between the student’s prior year and 
current year scores on state assessments 
in English language arts, math, science, 
and social science. 

Based on the change in the school's 
absolute performance index rating from the 
prior year to the current year. 

a. The percentage tested at the school, which is used in the calculation of the Elementary and Secondary 
Education Act performance index score, is not used in this study because preliminary analyses revealed both 
measures to be highly skewed due to the fact that most schools met the target. 

b. Although graduation rate weighted points is a measure in the Elementary and Secondary Education Act per- 
formance index, it was used in this study as a measure of the absolute performance factor in the high school 
factor models along with the other graduation rate measures. This change in structure was made due to the 
high correlations between the graduation rate measures. 

Source: South Carolina Education Oversight Committee, 2011. 


Using three indices 
to evaluate school 
performance can 
be confusing to 
educators and 
other stakeholders. 
Although the 
conceptual 
distinctions can 
be explained, 
communicating 
results to 
educators is 
challenging when 
ranking of schools 
is inconsistent 
across the indices 


indices. While schools generally see consistent results between their absolute and ESEA 
performance index scores, their growth performance index scores are often unrelated or 
inversely related to the other two indices (which may be due in part to whether the growth 
performance index score is calculated as a within- or cross-cohort comparison). Empirical 
evidence of these inconsistencies can be seen in the correlations between index scores. 

The correlation between the growth and ESEA performance indices is near 0 and not 
significant at all school levels (-.03 at the elementary school level, .06 at the middle school 
level, and -.08 at the high school level; see tables B2-B4 in appendix B). The correlation 
between the growth and absolute performance indices is significant but weak (-.20 at the 
elementary school level, -.13 at the middle school level, and .20 at the high school level). 
The correlation between the absolute and ESEA indices is significant and ranges from 
.72 (at the high school level) to .76 (at the elementary and middle school levels), which 
suggests that high scores on one index do not perfectly align to high scores on the other. 

Correlations can be useful when explaining why schools’ performance ratings vary. The 
square of a correlation coefficient is the amount of variance in one measure explained 
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by the other. For the absolute and ESEA performance indices, the .72—76 correlations 
translate into a variance of .52—58. This means that 52 percent of the reasons why schools 
differ on the absolute performance index can be explained when schools’ ESEA perfor- 
mance index scores are known. The implication is that the South Carolina Department 
of Education can understand some of the reasons why schools vary on the absolute perfon 
mance index (based on the ESEA performance index) but that 48 percent of the reasons 
why schools differ is unaccounted for. The growth performance index in elementary school 
may explain 4 percent of the reasons why schools differ in the absolute performance index 
(that is, - 20 A 2), but this represents a very small amount. 


Several methods exist to combine scores from the measures that make up the three existing 
indices to an overall index of school performance, such as taking their mean or summing 
them. While using simple mathematical operations is appealing for practical reasons, 
several challenges to their use exist. For example, the indices are on different scales, which 
could cause some indices to have greater influence in the calculation of the overall index. 
Also, in a simple additive model, strengths on one index may mask weaknesses on another 
index unless a weighting system is applied. This study uses an alternative approach called 
confirmatory factor analysis to evaluate how an overall score might be created (see box 1 
for definitions of key terms in the report). 

To meet the needs of the South Carolina Department of Education, this study addresses 
three research questions: 

• Can the measures that make up each of the three indices of school performance in 
South Carolina be combined into an overall, reliable alternative index of school 
performance? 
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Box 1. Key terms 

Beating the odd schools. Schools whose performance scores are better than expected, after 
school demographic characteristics are controlled for. These schools can be identified by com- 
paring their school performance score with their expected performance based on their demo- 
graphic characteristics (race/ethnicity, gender, disability status, and economic status). 

Confirmatory factor analysis. A statistical model used to estimate the relationship between 
observed measures (for example, scores on several different reading tests) and an unobserved 
construct (for example, school quality) that is believed to underlie the observed measures. The 
unobserved construct is called a factor or latent variable when it is estimated using two or 
more observed measures. 

Goodness of fit. The relationship between observed data and what is predicted by a specified 
model. The relationship can be tested by a set of different goodness-of-fit indices. Index values 
that fall into an accepted range are viewed as evidence of a strong relationship between the 
observed data and what is predicted by the model. Subsequently, the model is deemed to 
demonstrate “good fit.” 

Latent variable. The underlying construct (for example, school quality) that the observed vari- 
ables are expected to measure. Latent variables are unobserved and cannot be measured 
directly and therefore can only be estimated statistically based on available data. 
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• Which schools have observed school performance scores on the new index that 
are better than expected (that is, beating the odds) after school demographic char- 
acteristics are controlled for? 

• Can South Carolina schools be categorized by their demographic profiles, based 
on their predominant student demographic characteristics, and can the demo- 
graphic profiles of schools that are beating the odds be identified? 

In addition to the confirmatory factor analysis discussed above, which is used to answer 
the first two research questions, the study uses latent profile analysis to identify the 
demographic profiles of South Carolina schools and to determine which profiles include 
high-performing schools (the third research question). 


The study uses data provided by the South Carolina Department of Education on each 
elementary, middle, and high school in the state for 2012/13, including total number of 
students; number of students who are White, Black, and Hispanic; number of male and 
female students; number of disabled students; poverty index (South Carolina’s measure of 
school poverty); and results for each of the three school performance indices and for the 
measures that make up the indices. The data used in the study are also available on the 
South Carolina Department of Education website as a component of school report cards. 
See appendix A for details on South Carolina’s current school performance indices and 
appendix B for details on the analyses. 
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What the study found 

This section describes the results of the analyses used to derive an alternative school perfor- 
mance index, identify high-performing schools relative to observable similar schools (that 
is, beating the odds schools), and describe the demographic characteristics of those schools. 


The three school performance indices currently used in South Carolina to rate schools can be 
combined into an overall, reliable alternative index of school performance using a bi-factor model 

This study compares four confirmatory factor analysis models at the elementary, middle, 
and high school levels to test different specifications of whether a singular construct (in this 
case, an overall, reliable index of school performance) exists based on the available data (in 
this case, the measures from existing school performance indices in South Carolina). 

At the elementary and middle school levels, the first model is a one-factor model that 
hypothesizes that only one school performance construct exists and can be measured by all 
the status and growth measures currently used by the South Carolina Department of Edu- 
cation at each school level (see figure B1 in appendix B). The second model is a two-factor 
model that tests the possibility that there are two distinct, yet related, constructs — one 
(annual school performance) that can be indicated by multiple status measures and one 
(growth performance) that can be indicated by one or more growth measures (see figure 
B2 in appendix B). The third model is a three-factor model that tests the possibility that 
there are three distinct but related constructs; the constructs in this model are consistent 
with the three indices currently used by the South Carolina Department of Education 
(that is, absolute performance, ESEA subgroup performance, and growth performance) 
except that the weights for each measure differ from those used by the department (see 
figure B3 in appendix B). The fourth model is a bi-factor model that theorizes that absolute 
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performance, ESEA subgroup performance, and growth performance each exist as specific 
constructs but that an additional, general construct of school performance exists that cap- 
tures something in common across all of the measures (see figure B4 in appendix B). 

The four models tested at the high school level differed slightly from the models tested at the 
elementary and middle school levels because there is only one growth measure at the high 
school level. The one-factor model shares the same structure as the elementary and middle 
school models (see figure B5 in appendix B); the other models omit a separate growth per- 
formance factor and substitute an observed measure. Therefore, the second model is also a 
oneTactor model, but it tests whether there is a distinct construct (annual school performance) 
that is separate but related to the growth measure (see figure B6 in appendix B). The third 
model is a twoTactor model that tests the possibility that there are two distinct but related 
constructs that are also related to the growth measure (see figure B7 in appendix B). The 
fourth model is a bi-factor model that tests the addition of a general construct of school per- 
formance that captures what is common across all the measures (see figure B8 in appendix B). 

Testing these four models at each level allowed the study team to determine whether 
having one construct or multiple constructs best characterizes the data. An advantage of 
the bhfactor model is that it allowed the study team to account for the possibility of multi- 
ple constructs while providing an overall score from the general factor that could be used 
as the global index. Other key differences between confirmatory factor analysis models are 
noted in box B1 in appendix B. 

The four confirmatory factor analysis models were compared using different goodness-of- 
fit indices to determine which one best explained the relationship among the observed 
measures (Rindskopf & Rose, 1988). Across all school levels the bi-factor model was found 
to provide the best fit to the data (see table B8 in appendix B for summary of fit indices). 
Figure 1 presents the bi-factor model used at the elementary and middle school levels, and 
figure 2 presents the bi-factor model used at the high school level. 

While the bi-factor model results in up to four separate factor scores (depending on the 
model) — representing the constructs of absolute performance, ESEA performance, growth 
performance, and school performance — the interest of this study lies in the common vari- 
ance explained by the school performance general factor. At the elementary and middle 
school levels the school performance general factor explains much of the common vari- 
ance between the measures that represent absolute and ESEA performance but little of the 
variance in the growth measures (see table B9 in appendix B). A similar pattern exists at 
the high school level (see table BIO in appendix B), based on the single observed growth 
measure. This result suggests that the common variance found in the growth measures is 
not explained by the school performance general factor. 

Despite the limitations of the growth measures, the school performance general factor 
effectively combines all the observed measures into one index that can be used to evalu- 
ate school performance and is thus used in subsequent statistical analysis as an outcome 
of school performance for the analysis in the next section. The reliability of the school 
performance general factor was estimated at .89 at the elementary school level, .90 at the 
middle school level, and .95 at the high school level (see appendix B for details on compu- 
tation). Reliability of at least .80 is considered acceptable for research purposes, and values 
of at least .90 are appropriate for clinical decisionmaking. 
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Figure 1. Sample South Carolina bi-factor model specification at the elementary and middle school levels 



ESEA is Elementary and Secondary Education Act, WELAIS is writing and English language arts combined index score, MIS is math 
index score, SCIS is science index score, SSIS is social science index score, ELAPA is English language arts weighted points, MATHPA 
is math weighted points, SCPA is science weighted points, SSPA is social science weighted points, ELAG is English language arts 
growth index, MATHG is math growth index, SCG is science growth index, and SSG is social science growth index. 

Note: Elementary school refers to grades 3-5, and middle school refers to grades 6-8. 

Source: Authors' illustration. 


Figure 2. Sample South Carolina bi-factor model specification at the high school level 



ESEA is Elementary and Secondary Education Act, LHSAP is longitudinal (spring) High School Assessment Program passage rate, HSAP 
is first attempt High School Assessment Program passage rate, EOC is end-of-course exam overall passage rate, ONTIME is four-year 
cohort graduation rate, FIVEYR is five-year graduation rate, GRAD is graduation rate weighted points, ELAPA is English language arts 
weighted points, MATHPA is math weighted points, SCPA is science weighted points, SSPA is social science weighted points, and 
GROWTH is change in absolute performance index from prior year. 

Note: High school refers to grades 9-12. 

Source: Authors' illustration. 
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The good fit of the bi-factor model and the high reliability of the school performance 
general factor mean that all the measures used in the three existing indices can be com- 
bined into an overall, reliable alternative index of school performance. 

Under the alternative index of school performance, about 3 percent of elementary schools, 

2 percent of middle schools, and 3 percent of high schools beat the odds, given the schools’ 
demographic characteristics 

Schools beating the odds can be identified by comparing their score on the school per- 
formance general factor with how they would be expected to perform based on their 
demographic characteristics (that is, race/ethnicity, gender, disability status, and economic 
status). A school’s expected performance can be predicted after the relationship between 
a school’s performance factor score and a set of variables (in this case demographic char- 
acteristics) has been determined using an accepted statistical approach (see appendix B). 

The school performance general factor scores were calculated on a scale that has a mean 
of 0 and a standard deviation of 1. Thus, schools with a school performance general factor 
score of 0 are considered average, schools with positive school performance general factor 
scores are considered above average, and schools with negative school performance general 
factor scores are considered below average. 

Positive differences between observed and expected scores indicate better than expect- 
ed performance. In 2012/13 the difference (called a residual) between the observed and 
expected school performance general factor score ranged from -1.60 to 1.78 at the elemen- 
tary school level (residual standard error of ±.53), from -1.88 to 1.60 at the middle school 
level (residual standard error of ±.54), and from -1.89 to 1.77 at the high school level (resid- 
ual standard error of ±.49). 

Of the 304 elementary schools with a positive residual, 18 (about 3 percent of elementary 
schools statewide) had a difference large enough to be considered reliable (see appendix B 
for a discussion of the 95 percent confidence interval). These 18 schools beat the odds in 
school performance during the 2012/13 school year, with higher school performance than 
would be predicted after the demographic characteristics of their students were controlled 
for; the residuals of these schools ranged from 1.05 to 1.78. Of the 146 middle schools with 
a positive residual, 6 (about 2 percent) were identified as high performing relative to their 
expected school performance general factor scores after the schools’ demographic char- 
acteristics were controlled for. The residuals of these schools ranged from 1.03 to 1.60. Of 
the 105 high schools with a positive residual, 6 (about 3 percent) were identified as high 
performing, with residuals that ranged from .96 to 1.77. 

The number of school demographic profiles identified by latent profile analysis differs by school 
level, and most profiles contained at least one school that was beating the odds 

To determine the extent to which high-performing schools were demographically similar 
to other schools in South Carolina, the study team used latent profile analysis — which 
allows for the identification of subgroups within multivariate data and is a special case of 
cluster analysis — to identify distinct school demographic profiles in the state. Based on 
the average demographic characteristics of each profile, schools with similar demographics 
can be matched in a “nearest neighbor” approach. For example, a school with a lower than 


Schools beating 
the odds can 
be identified by 
comparing their 
score on the school 
performance 
general factor with 
how they would 
be expected to 
perform based on 
their demographic 
characteristics. 
Positive differences 
between 
observed and 
expected scores 
indicate better 
than expected 
performance 
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expected school performance general factor score might be paired with a school within the 
same profile that has a higher than expected school performance general factor score (that 
is, a school beating the odds). Such a pairing would allow for successful schools to mentor 
struggling schools with similar demographic characteristics. 

Each profile is named for its predominant student demographic characteristic or charao 
teristics, with the term “high” used when the percentage of students in the category was 
above the state average and exceeded 70 percent. References to the state “average” (that 
is, below, above, or at the average) are used to further distinguish the profile from other 
profiles and are generally qualitative judgments. 


Elementary schools. Five distinct demographic profiles were identified at the elementary 
school level (table 2): 

• Profile 1: High percentage of White students and average poverty index (35 percent 
of schools). 

• Profile 2: High percentage of White students and below average poverty index 
(13 percent of schools). 

• Profile 3: General population (22 percent of schools). This profile reflects the 
approximate state averages for all characteristics. 

• Profile 4: High poverty index and above average percentage of Hispanic students 
(10 percent of schools). 

• Profile 5. High percentage of Black students and high poverty index (20 percent of 
schools). 


Five distinct 
demographic 
profiles were 
identified at 
the elementary 
school level; each 
profile included 
at least one of the 
18 elementary 
schools identified 
by the alternate 
index of school 
performance as 
beating the odds 


Table 2. Five demographic profiles of South Carolina public elementary schools, 2012/13 (mean 


percentages, unless otherwise noted) 

Demographic 

characteristic 

State 

profile 

Profile 1: 
High 

percentage of 
White students 
and average 
poverty index 

Profile 2: 
High 

percentage 
of White 
students and 
below average 
poverty index 

Profile 3: 
General 
population 

Profile 4: 
High poverty 
index and 
above average 
percentage 
of Hispanic 
students 

Profile 5: 
High 

percentage of 
Black students 
and high 
poverty index 

White 

49.27 

70.73 

75.15 

41.74 

36.20 

9.50 

Black 

41.10 

20.03 

15.99 

48.60 

38.00 

85.89 

Hispanic 

6.93 

6.26 

4.51 

5.74 

23.64 

2.97 

Male 

51.45 

51.45 

51.20 

51.59 

51.77 

51.31 

Disabled 

12.57 

12.75 

10.07 

13.65 

12.37 

12.81 

Income status 

Mean poverty index 

75.29 

70.96 

36.64 

82.35 

84.55 

95.62 

Number and percentage of schools 

Number 

619 

215 

81 

139 

62 

122 

Percent 

na 

35 

13 

22 

10 

20 

School performance 

Mean school performance index 

0.04 

0.40 

1.35 

-0.19 

-.37 

-1.01 


na is not applicable. 

Note: Elementary school refers to grades 3-5. 

Source: Authors' analysis based on data from the South Carolina Department of Education (2014a). 
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Figure 3. All five demographic profiles of elementary schools in South Carolina 
included at least one school that was beating the odds based on its rating under 
the alternative school performance index, 2012/13 


Number of beating the odds schools 



Profile 1: Profile 2: 

High percentage High percentage 
of White students of White students 
and average and below average 
poverty index poverty index 


Profile 3: 
General 
population 


Profile 4: 

High poverty index 
and above average 
percentage of 
Hispanic students 


Profile 5: 

High percentage 
of Black students 
and high 
poverty index 


Note: n = 18. Elementary school refers to grades 3-5. 

Source: Authors' analysis based on data from the South Carolina Department of Education (2014a). 


Each profile included at least one of the 18 elementary schools identified in the previous 
section as beating the odds (figure 3). Profiles 1, 3, and 5 each had five schools that were 
beating the odds; profiles 2 and 4 had fewer. It is unsurprising that the schools beating the 
odds are fairly evenly distributed across the different profiles because the individual school 
residuals used to identify schools as beating the odds are uncorrelated with the demo- 
graphic characteristics of each school. 


Four distinct 
demographic 
profiles were 
identified at the 
middle school 
level; three profiles 
included at least 
one of the six 
middle schools 
identified by 
the alternative 
index of school 
performance as 
beating the odds 


Middle schools. Four distinct demographic profiles were identified at the middle school 
level (table 3): 

• Profile 1: High percentage of White students and below average poverty index 
(12 percent of schools). 

• Profile 2: High percentage of Black students and high poverty index (23 percent of 
schools). 

• Profile 3: High poverty index and above average percentage of Hispanic students 
(10 percent of schools). 

• Profile 4: General population (55 percent of schools). This profile reflects the 
approximate state averages for all characteristics. 


Profiles 1, 2, and 4 included at least one of the six middle schools identified in the previous 
section as beating the odds; profile 3 did not (figure 4). 


High schools. Three distinct demographic profiles were identified at the high school level 
(table 4): 

• Profile 1: High percentage of White students and below average poverty index 
(11 percent of schools). 
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Table 3. Four demographic profiles of South Carolina public middle schools, 
2012/13 (mean percentages, unless otherwise noted) 


Demographic 

characteristic 

State 

profile 

Profile 1: 
High 

percentage 
of White 
students 
and below 
average 
poverty index 

Profile 2: 
High 

percentage 
of Black 
students 
and high 
poverty index 

Profile 3: 
High poverty 
index and 
above 
average 
percentage 
of Hispanic 
students 

Profile 4: 
General 
population 

White 

50.62 

75.08 

13.71 

43.15 

62.15 

Black 

42.02 

16.68 

81.61 

40.99 

30.47 

Hispanic 

4.64 

4.27 

2.20 

12.86 

4.22 

Male 

51.03 

49.53 

50.83 

51.64 

51.34 

Disabled 

12.93 

7.68 

15.35 

13.14 

13.01 

Income status 

Mean poverty index 

73.66 

35.70 

93.86 

79.67 

72.20 

Number and percentage of schools 

Number 

286 

35 

66 

29 

156 

Percent 

na 

12 

23 

10 

55 

School performance 

Mean school performance index 

0.02 

1.47 

-.91 

-0.15 

0.12 


na is not applicable. 

Note: Middle school refers to grades 6-8. 

Source: Authors' analysis based on data from the South Carolina Department of Education (2014a). 


Figure 4. Demographic profiles 1, 2, and 4 of middle schools in South Carolina 
included at least one school that was beating the odds under the alternative 
school performance index, 2012/13 


Number of beating the odds schools 



Profile 1: 

High percentage 
of White students 
and below average 
poverty index 


Profile 2: 

High percentage 
of Black students 
and high 
poverty index 


Profile 3: 

High poverty index 
and above average 
percentage of 
Hispanic students 


Profile 4: 
General 
population 


Note: n = 6. Middle school refers to grades 6-8. 

Source: Authors' analysis based on data from the South Carolina Department of Education (2014a). 
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Table 4. Three demographic profiles of South Carolina public high schools, 
2012/13 (mean percentages, unless otherwise noted) 


Demographic 

characteristic 

State 

profile 

Profile 1: 

High percentage 
of White students 
and below average 
poverty index 

Profile 2: 
General 
population 

Profile 3: 

High percentage 
of Black students 
and high 
poverty index 

White 

50.75 

74.24 

60.71 

14.44 

Black 

42.79 

17.52 

31.88 

80.61 

Hispanic 

4.16 

5.05 

4.58 

2.74 

Male 

50.83 

50.07 

51.01 

50.76 

Disabled 

12.20 

7.90 

12.01 

14.63 

Income status 

Mean poverty index 

69.88 

35.25 

67.71 

91.13 

Number and percentage of schools 

Number 

201 

23 

128 

50 

Percent 

na 

11 

64 

25 

School performance 

Mean school performance index 

0.06 

1.17 

0.27 

-0.96 


na is not applicable. 

Note: High school refers to grades 9-12. 

Source: Authors' analysis based on data from the South Carolina Department of Education (2014a). 


Figure 5. Demographic profiles 2 and 3 of high schools in South Carolina included 
at least one school that was beating the odds under the alternative school 
performance index, 2012/13 


Number of beating the odds schools 



Profile 1: Profile 2: Profile 3: 

High percentage General High percentage 

of White students population of Black students 

and below average and high 

poverty index poverty index 


Note: n = 6. High school refers to grades 9-12. 

Source: Authors' analysis based on data from the South Carolina Department of Education (2014a). 
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• Profile 2: General population (64 percent of schools). This profile reflects approxi- 
mate state averages for all characteristics. 

• Profile 3: High percentage of Black students and high poverty index (25 percent of 
schools). 

Profiles 2 and 3 included at least one of the six high schools identified in the previous 
section as beating the odds; profile 1 did not (figure 5). 

Implications, limitations, and next steps 


This study demonstrates that South Carolina could combine the existing three indices 
of school performance into one index, thereby reducing the burden on parents and the 
public in interpreting three indices with inconsistent rankings. Beyond its use in a school 
accountability framework, the combined index could also be used in South Carolina to 
identify schools that are beating the odds. In addition, the study results could inform deci- 
sions related to the development of new accountability indices in other states with similar 
models. 


Three distinct 
demographic 
profiles were 
identified at 
the high school 
level; two profiles 
included at least 
one of the six high 
schools identified 
by the alternate 
index of school 
performance as 
beating the odds 


Several limitations of the study should be considered. First, the analyses were conducted 
using school-level data instead of student-level data. Using student-level data would permit a 
more rigorous evaluation by allowing for the modeling of hierarchical data (that is, students 
within classes within schools) and the characteristics of individual students. Additionally, 
the school performance estimates and ranking of schools depend on the schools included in 
the analysis. While outliers were removed, there may be schools in the sample that serve a 
specialized student population based on specified enrollment criteria (for example, a charter 
school serving high-performing students or a school whose population consists primarily 
of students with disabilities). Such schools have historically demonstrated higher or lower 
performance on statewide assessments, and thus including those schools may affect which 
schools are ultimately identified as high performing. Future work should include a sensitivity 
analysis to evaluate how the school performance estimates change as particular schools are 
excluded, as well as which schools may be differentially identified as high performing. 

In addition, the measures that are currently used by the South Carolina Department of 
Education to indicate growth could be studied further. While the bi-factor model provid- 
ed the best fit to the data, the common variance found in the growth measures was not 
explained by the school performance general factor. This means that the growth measures, 
for the most part, are not contributing to the estimation of the school performance general 
factor. This is especially true at the high school level, where there is only one measure of 
growth. As explained in appendix A, the measure of growth at the high school level is 
the difference between the current year and prior year’s absolute performance index at the 
school. This difference measure could be studied further to better understand its contribu- 
tion to explaining school performance in South Carolina. 

Finally, only one year of school performance data was used in the analyses. Although the 
current results provide preliminary support for the use of a global school performance 
index, it is important to test the invariance of the study findings over several years. By 
using the factor loadings, residual variances, and error covariances in this study, the South 
Carolina Department of Education could both replicate the analyses reported here and 
extend the model to test for how well the approach replicates over time. 
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Appendix A. South Carolina’s current school performance indices 


South Carolina rates school performance using three indices: an absolute performance 
index, an index developed to comply with the Elementary and Secondary Education Act 
(ESEA), and a growth performance index. Each index is described in detail below. 

Absolute performance index 

Elementary and middle schools. All schools serving grades 3-8 receive an absolute per- 
formance index rating. Each school’s rating is based on the number of students who meet 
a given performance level on the state assessment, the Palmetto Assessment of State Stan- 
dards (PASS), which covers reading and research (referred to as English language arts), 
writing, math, social studies, and science. PASS scores are reported on a within-grade scale 
of 300-900, and the scale score needed to meet gradedevel performance standards is 600 
(South Carolina Department of Education, 2014b). 

In addition to the PASS, some students in grades 6-8 also have scores on end-of-course 
exams in algebra I, English I, and physical science, which are typically taken in high 
school. The scores on these assessments are included in the rating of the school where the 
student is enrolled. This means that the elementary school ratings include five subjects, 
but the middle school ratings can include up to eight, depending on whether the student 
has taken one or more end-of-course exams. End-of-course exam scores are reported on a 
scale of 0-100 (South Carolina Department of Education, 2014c). 

The state has designated five proficiency levels for the PASS, and each proficiency level is 
worth between one and five points on the absolute performance index. All the subject area 
tests use the same proficiency categories and assign the same point values to each level, though 
the outpoints for each proficiency category vary by grade level and subject area. In addition, 
schools earn points based on the scores of students on all three possible end-of-course exams. 

The absolute performance index ratings for elementary and middle schools (grades 3-8) 
are calculated in four steps: 

1. Calculate the average points per subject per school by multiplying the points assigned 
to each PASS proficiency level or end-of-course exam score (table Al) and dividing 
by the total number of tests. For grades with a writing test, the English language arts 
index is a composite of the PASS English language arts (67 percent) and the PASS 
writing (33 percent). 

2. Weight each subject area based on state statute by multiplying the subject area indices 
by the approved weight (table A2). 2 These weighted values serve as the observed mea- 
sures used in the confirmatory factor analysis detailed in appendix B. 

3. Sum the unrounded weighted values to create an index score that is then rounded to 
the nearest hundredth (see table A3 for an example of this step). 

4- Assign the final absolute performance index rating using the approved scale (table A4). 
(For the hypothetical school in table A3, the score of 3.14 would earn an “average” 
absolute rating.) 
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Table Al. Rating points received for each Palmetto Assessment of State Standards 
proficiency level and end-of-course grade at the elementary and middle school 
levels under South Carolina’s absolute performance index, 2012/13 


Palmetto Assessment of State 
Standards proficiency level 

End-of-course grade 

Rating points 

Exemplary 5 

A 

5 

Exemplary 4 

B 

4 

Met 

C 

3 

Not met 2 

D 

2 

Not met 1 

F 

1 

Note: Elementary school covers grades 3-5, and middle school covers grades 6-8. 
Source: South Carolina Education Oversight Committee, 2011. 


Table A2. Subject area weights at the elementary and middle school levels under 
South Carolina’s absolute performance index, 2012/13 


School level 

English 
language arts 

Math 

Science 

Social 

studies 

Elementary (grades 3-5) 

0.30 

0.30 

0.20 

0.20 

Middle (grades 6-8) 

0.25 

0.25 

0.25 

0.25 


Source: South Carolina Education Oversight Committee, 2011. 


Table A3. Example rating calculation for an elementary school under South Carolina’s absolute 
performance index, 2012/13 

Palmetto Assessment of 
State Standards (PASS) 
proficiency level or 
calculation component 

English 
language arts 

Math 

Science 

Social 

studies 

Rating 

index 

PASS proficiency level 

Exemplary 5 

26 

22 

14 

18 

na 

Exemplary 4 

35 

37 

25 

28 

na 

Met 

56 

64 

28 

33 

na 

Not met 2 

43 

32 

23 

14 

na 

Not met 1 

15 

11 

14 

8 

na 

Calculation component 

Total points 

539 

525 

314 

337 

na 

Total tests 

175 

166 

104 

101 

na 

Index 

3.08 

3.16 

3.02 

3.34 

na 

Weight 

0.3 

0.3 

0.2 

0.2 

na 

Weighted value 

0.92 

0.95 

0.60 

0.67 

3.14 


na is not applicable. 

Note: Elementary school covers grades 3-5, and middle school covers grades 6-8. 
Source: Authors' illustration based on points and weights outlined in tables Al and A2. 
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Table A4. Rating scale under South Carolina’s absolute performance index, 2012/13 

I Score 

Rating i 

3.40 or higher 

Excellent 

3.18-3.39 

Good 

2.65-3.17 

Average 

2.32-2.64 

Below average 

2.31 or lower 

At risk 

Source: South Carolina Education Oversight Committee, 2011. 


High schools. Calculating ratings at the high school (grades 9-12) level involves the same 
steps as the elementary and middle school level but uses different measures. Instead of 
using PASS and end-of-course exam scores, the high school ratings are based on: 

• Longitudinal (spring) High School Assessment Program passage rate. 

• First attempt High School Assessment Program passage rate. 

• End'of'Course exam overall passage rate. 

• On-time graduation rate. 

• Five-year graduation rate. 

Each criterion is assigned a point value ranging from 1-5 based on the school’s score (table 
A5). 

The high school ratings aggregate student data to the school level before assigning points. 
Essentially the school receives points according to table A5 for the percentage of students 
who pass or graduate. These points are multiplied by the weight for that criterion (table 
A6) and then summed. The conversion from score to rating in table A4 is also used for 
high schools. 

Elementary and Secondary Education Act performance index 

South Carolina was awarded an ESEA waiver that allows the state to implement its own 
accountability system. The state developed a model that awards up to one point for each 
subgroup that meets the target for a certain measure, such as a state assessment or other 
outcome. The targets, or annual measurable objectives, are described in table A7. 


Table A5. Cutpoints for high school ratings under South Carolina’s absolute performance index, 
2012/13 (percent) 

1 Criterion 

1 point 

2 points 

3 points 

4 points 

5 points 

Longitudinal (spring) High School Assessment 
Program passage rate 

75.8 or lower 

75.9-84.0 

84.1-94.2 

94.3-96.9 

97.0 or higher 

First attempt High School Assessment 
Program passage rate 

53.1 or lower 

53.2-63.0 

63.1-82.9 

83.0-92.9 

93.0 or higher 

End-of-course exam overall passage rate 

30.7 or lower 

30.8-41.9 

42.0-64.2 

64.3-75.4 

75.5 or higher 

On-time graduation rate 

47.3 or lower 

47.4-59.5 

59.6-83.9 

84.0-96.0 

96.1 or higher 

Five-year graduation rate 

50.2 or lower 

50.3-62.6 

62.7-87.6 

87.7-96.9 

97.0 or higher 

Note: High school refers to grades 9-12. 

Source: South Carolina Education Oversight Committee, 2011. 
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Table A6. Subject area weights under South Carolina’s high school absolute 

performance index, 2012/13 


1 Criterion 

Weight S 

Longitudinal (spring) High School Assessment Program passage rate 

.20 

First attempt High School Assessment Program passage rate 

.20 

End-of-course exam overall passage rate 

.20 

On-time graduation rate 

.30 

Five-year graduation rate 

.10 

Source: South Carolina Education Oversight Committee, 2011. 


Table A7. Targets under South Carolina’s Elementary and Secondary Education Act 
performance index, by school level, 2012/13 

Measure 

Elementary schools 
(grades 3-5) 

Middle schools 
(grades 6-8) 

High schools 
(grades 9-12) 

English language arts 
proficiency 

Mean PASS 
score of 635 

Mean PASS 
score of 628 

Mean High School 
Assessment Program 
score of 226 

Math proficiency 

Mean PASS 
score of 635 

Mean PASS 
score of 628 

Mean High School 
Assessment Program 
score of 223 

Science proficiency 

Mean PASS 
score of 635 

Mean PASS 
score of 628 

Mean end-of-course 
exam score of 77 

Social studies and history 
proficiency 

Mean PASS 
score of 635 

Mean PASS 
score of 628 

Mean end-of-course 
exam score of 73 

Percentage of students tested 
in English language arts 

95 

95 

95 

Percentage of students tested 
in math 

95 

95 

95 

Met or improved graduation rate 

na 

na 

74.1 


PASS is Palmetto Assessment of State Standards, 
na is not applicable. 

Note: The High School Assessment Program scores range from 100 to 320 (South Carolina Department of 
Education, 2014d). 

Source: South Carolina Department of Education, 2013. 


For a given outcome there are up to 10 demographic subgroups (male students, female stu- 
dents, White students, Black students, Asian/Pacific Islander students, Hispanic students, 
American Indian/Alaskan students, disabled students, limited English proficient students, 
and students eligible for free or reduced-price lunch * 1 2 3 ) in addition to the all students group; 
typically one or more are excluded because of small numbers of students. The specific mea- 
sures and their weighted values are shown in table A8. 

A school’s rating is calculated in three steps: 

1. Sum the total points on the measure earned by all the available subgroups. 

2. Divide the total points by the number of available subgroups to generate the percent- 
age of subgroups meeting the target. 
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Table A8. Weights under South Carolina’s Elementary and Secondary Education Act 
performance index, by school level, 2012/13 


Elementary and middle schools 

High schools 

Measure 

(grades 3-8) 

(grades 9-12) 

English language arts proficiency 

.35 

.225 

Math proficiency 

.35 

.225 

Science proficiency 

.05 

.05 

Social studies and history proficiency 

.05 

.05 

Percentage of students tested in 
English language arts 

.10 

.075 

Percentage of students tested in math 

.10 

.075 

Met or improved graduation rate 

na 

.30 

na is not applicable. 

Source: South Carolina Department of Education, 2013. 



3. Multiply the percentages by the weights in table A8 and sum the total points. This 
produces an index score between 0 and 100, which is used to assign a letter grade from 
A to F, where 90-100 is an A, 80-89.9 is a B, and so forth. The individual subject area 
components serve as the observed measures used in the confirmatory factor analysis 
detailed in appendix B. 

Growth performance index 

Elementary schools. The growth performance index rating at the elementary and middle 
school levels is based on value tables and studentdevel growth within cohorts. A value 
table assigns a point value based on the change between a student’s prior and current year 
scores on the PASS (table A9). For example, a student who scored “not met 1” in one year 
and “not met 2” in the next year would earn 100 points, and a student who scored “met” in 
one year and “exemplary 4” in the next year would earn 110 points. 

A hypothetical distribution for an elementary school is shown in table A10. 

Table All shows the points earned for school X based on the number of points received for 
each combination of prior and current year PASS scores (see table A9) multiplied by the 
number of students who achieved each combination (see table A10). The average score for 
school X’s 382 students is 95.92, which resolves to a rating of “average” (table A12). 


Table A9. Value table for calculating ratings for elementary and middle schools 
under South Carolina’s growth performance index, 2012/13 

Prior year Palmetto 

Current year Palmetto Assessment of State Standards proficiency level 

Assessment of State 
Standards proficiency level 

Not met 1 

Not met 2 

Met 

Exemplary 4 

Exemplary 5 

Exemplary 5 

40 

60 

80 

90 

100 

Exemplary 4 

50 

70 

90 

100 

110 

Met 

60 

80 

100 

110 

120 

Not met 2 

70 

90 

110 

120 

130 

Not met 1 

80 

100 

120 

130 

140 

Note: Elementary school covers grades 3-5, and middle school covers grades 6-8. 


Source: South Carolina Education Oversight Committee, 2011. 









Table A10. Hypothetical elementary school distribution of prior and current year 
Palmetto Assessment of State Standards proficiency levels: School X 


Prior year Palmetto 
Assessment of State 
Standards proficiency level 


Current year Palmetto Assessment of 
State Standards proficiency level 


Not met 1 Not met 2 


Met 


Total 

number of 

Exemplary 4 Exemplary 5 students 


Exemplary 5 
Exemplary 4 
Met 

Not met 2 
Not met 1 


0 4 15 22 21 62 

1 4 22 26 17 70 

8 26 38 24 15 111 

15 28 20 11 1 75 

27 17 14 6 0 64 


Note: n = 382. 

Source: Authors' illustration. 


Table All. School X value table result 


Prior year Palmetto 
Assessment of State 
Standards proficiency level 


Current year Palmetto Assessment of 
State Standards proficiency level 


Total points 
for each 

Not 
met 1 

Not 
met 2 

Met 

Exemplary 4 Exemplary 5 

proficiency 
level in 
prior year 

Exemplary 5 

0 

240 

1,200 

1,980 

2,100 

5,520 

Exemplary 4 

50 

280 

1,980 

2,600 

1,870 

6,780 

Met 

480 

2,080 

3,800 

2,640 

1,800 

10,800 

Not met 2 

1,050 

2,520 

2,200 

1,320 

130 

7,220 

Not met 1 

2,160 

1,700 

1,680 

780 

0 

6,320 

Total points across all 






36,640 

combinations of proficiency 







levels 

na 

na 

na 

na 

na 


Average (n = 382) 

na 

na 

na 

na 

na 

95.92 

na is not applicable. 


Source: Authors’ illustration based on values and distribution of proficiency levels outlines in tables A9 and A10. 


Table A12. Rating cutpoints for elementary and middle schools (grades 3-8) under 

South Carolina’s growth performance index, 2012/13 


Index score 

Rating 

98.48 or higher 

Excellent 

96.39-98.47 

Good 

92.20-96.38 

Average 

90.11-92.19 

Below average 

90.10 or lower 

At risk 

Source: South Carolina Education Oversight Committee, 2011. 
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High schools. The approach to calculating the rating at the high school level under the 
growth performance index is greatly simplified from the approach used at the elementary 
and middle school levels. The high school rating is simply the change in the school’s abso- 
lute rating from the prior year to the current year. This means the growth is based on the 
difference in the absolute score from two different cohorts of students (that is, cross-cohort 
growth). The cutpoints are noted in table A13. 


Table A13. Rating cutpoints for high schools under South Carolina’s growth 
performance index, 2012/13 

j| Index score 

Rating | 

0.4 or greater 

Excellent 

0.3 

Good 

0.1-0. 2 

Average 

0.0 

Below average 

-0.1 or less 

At risk 

Note: High school covers grades 9-12. 

Source: South Carolina Education Oversight Committee, 2011. 
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Appendix B. Details on the analyses and results 


This study entailed a multistage analysis designed to develop an understanding of South 
Carolina’s current methodology for rating schools and explore the creation of an alterna- 
tive composite index that can also be used to identify schools beating the odds. In addi- 
tion, it conducted analyses to identify distinct school profiles in order to allow matching of 
schools with similar demographic characteristics. This appendix details how the analyses 
were conducted and provides results that supplement the findings in the main text. 

Preliminary analyses 

The study team first calculated means and standard deviations for the three current rating 
indices (absolute performance, Elementary and Secondary Education Act [ESEA] perfor- 
mance, and growth performance; table Bl), correlations between the current indices (tables 
B2-B4), and means and standard deviations for the individual measures that make up the 
indices as well as correlations between them (tables B5-B7). The correlation between the 
absolute and ESEA performance indices is high, ranging from .72 at the high school level 
to .76 at the elementary and middle school levels. The correlation between the growth 
performance index and the other two indices is low or nonsignificant at all levels. The 


Table Bl. Means and standard deviations for three South Carolina school ratings 
indices, by school level, 2012/13 


Index and statistics 

Elementary schools 
(grades 3-5) 

Vliddle schools 
(grades 6-8) 

High schools 
(grades 9-12) 

Absolute performance index 

Mean 

3.18 

3.05 

3.38 

Standard deviation 

.43 

.46 

.66 

Elementary and Secondary Education Act performance index 

Mean 

82.82 

77.52 

73.00 

Standard deviation 

17.40 

19.08 

17.74 

Growth performance index 

Mean 

100.65 

101.01 

.10 

Standard deviation 

1.61 

1.23 

.30 


Source: Authors' analysis based on data from the South Carolina Department of Education (2014a). 


Table B2. Correlations between South Carolina rating indices at the elementary 
school level, 2012/13 


Index 

Absolute 

performance 

index 

Elementary and 
Secondary 
Education Act 
performance index 

Growth 

performance 

index 

Absolute performance index 

1 



Elementary and Secondary Education 
Act performance index 

.76** 

1 


Growth performance index 

-.20** 

-.03 

1 


** Significant at the 0.01 level (two-tailed). 

Note: Elementary school covers grades 3-5. 

Source: Authors' analysis based on data from the South Carolina Department of Education (2014a). 
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Table B3. Correlations between South Carolina rating indices at the middle school 
level, 2012/13 


Index 

Absolute 

performance 

index 

Elementary and 
Secondary 
Education Act 
performance index 

Growth 

performance 

index 

Absolute performance index 

1 



Elementary and Secondary Education 
Act performance index 

.76" 

1 


Growth performance index 

-.13* 

.06 

1 


* Significant at the 0.05 level (two-tailed); ** significant at the 0.01 level (two-tailed). 

Note: Middle school covers grades 6-8. 

Source: Authors' analysis based on data from the South Carolina Department of Education (2014a). 


Table B4. Correlations between South Carolina rating indices at the high school 
level, 2012/13 


Index 

Absolute 

performance 

index 

Elementary and 
Secondary 
Education Act 
performance index 

Growth 

performance 

index 

Absolute performance index 

1 



Elementary and Secondary Education 
Act performance index 

.72" 

1 


Growth performance index 

.20" 

-.08 

1 


** Significant at the 0.01 level (two-tailed). 

Note: High school covers grades 9-12. 

Source: Authors' analysis based on data from the South Carolina Department of Education (2014a). 


same pattern exists in the relationships between the growth measures and the measures of 
the other two indices. 

Creating an overall, reliable alternative index of school performance using the measures that make 
up the three existing indices of school performance in South Carolina 

The purpose of research question 1 was to explore an alternative rating method that would 
allow South Carolina to create an overall school performance rating with existing mea- 
sures instead of using three different indices as it does now. 

Confirmatory factor analysis (Muthen & Muthen, 2012) was used to identify the measure- 
ment model that best explained the covariances among the observed measures. Four tra- 
ditional factor models were specified and differed slightly by school level: a unidimensional 
model of school performance indicated by all the observed school performance measures; 
a two-factor model with correlated constructs of annual performance and growth perfor- 
mance; a three-factor model with correlated constructs of absolute performance, growth 
performance, and ESEA subgroup performance; and a bi-factor model with individual con- 
structs of absolute performance, growth performance, and ESEA subgroup performance, as 
well as a school performance general factor that was indicated by all the variables (figures 
B1-B8). While the two- and three-factor models were estimated for comparison purposes, 
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Table B5. Correlations between and means and standard deviations for South Carolina rating index 
measures at the elementary school level, 2012/13 


Measure 

WELAIS 

MIS 

SCIS 

SSIS 

ELAPA 

MATHPA 

SCPA 

SSPA 

ELAG 

MATHG 

SCG 

SSG 

WELAIS 

1 












MIS 

.93** 












SCIS 

g2** 

.92** 

1 










SSIS 

.90** 

.88** 

.92* * 

1 









ELAPA 

.55** 

.52** 

.50** 

.49** 

1 








MATHPA 

.66** 

.76** 

.66** 

02** 

.54** 

1 







SCPA 

.59** 

.58** 

.66** 

.57** 

29** 

.45** 

1 






SSPA 

.61** 

52** 

.60** 

.70** 

.45** 

.61** 

.50** 

1 





ELAG 

—.17** 

—.17** 

-.19** 

-.19** 

-.03 

-.06 

.00 

-.06 

1 




MATHG 

—.12** 

-.07 

—.17** 

—.17** 

-.04 

.07 

.01 

-.04 

.63** 

1 



SCG 

—.11** 

-.08* 

-.03 

-.08 

-.01 

.01 

21** 

.05 

40** 

40** 

1 


SSG 

-.06 

-.04 

-.06 

.03 

.03 

.03 

.09* 

20** 

.39** 

.39** 

.36** 

1 

N 

649 

649 

649 

649 

645 

645 

645 

645 

639 

639 

634 

636 

Mean 

3.37 

3.18 

2.89 

3.21 

31.65 

25.57 

2.07 

3.60 

99.25 

101.00 

101.52 

102.07 

Standard 

deviation 

0.41 

0.48 

0.41 

0.51 

6.25 

11.20 

1.70 

1.52 

2.67 

3.03 

2.31 

2.84 


* Significant at the 0.05 level (two-tailed); ** significant at the 0.01 level (two-tailed). 

WELAIS is writing and English language arts combined index score, MIS is math index score, SCIS is science index score, SSIS is social 
science index score, ELAPA is English language arts weighted points, MATHPA is math weighted points, SCPA is science weighted 
points, SSPA is social science weighted points, ELAG is English language arts growth index, MATHG is math growth index, SCG is sci- 
ence growth index, and SSG is social science growth index. 

Note: Elementary school covers grades 3-5. 

Source: Authors’ analysis based on data from the South Carolina Department of Education (2014a). 
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Table B6. Correlations between and means and standard deviations for South Carolina rating index 
measures at the middle school level, 2012/13 


Measure 

WELAIS 

MIS 

SCIS 

SSIS 

ELAPA 

MATHPA 

SCPA 

SSPA 

ELAG 

MATHG 

SCG 

SSG 

WELAIS 

1 












MIS 

.93** 

1 











SCIS 

.90** 

.88** 

1 










SSIS 

.90** 

.85** 

.91** 

1 









ELAPA 

52** 

.58** 

.56** 

.58** 

1 








MATHPA 

.66** 

.74** 

.63** 

.63** 

.68** 

1 







SCPA 

.66** 

64** 

.76** 

Jl** 

.49** 

.56** 

1 






SSPA 

.65** 

.61** 

.66** 

.76** 

.57** 

.55** 

.66** 

1 





ELAG 

—.28** 

—.28** 

-.31** 

__24** 

.02 

-.13* 

-.17** 

-.16** 

1 




MATHG 

.02 

.09 

.00 

.07 

.07 

24** 

.10 

.05 

.45** 

1 



SCG 

—.27** 

__24** 

-.09 

-.16** 

.02 

-.05 

.16** 

-.04 

27** 

29** 

1 


SSG 

.01 

.00 

.03 

.12* 

.09 

.05 

.02 

.25** 

.36** 

24** 

.15** 

1 

N 

304 

304 

304 

304 

301 

301 

301 

301 

304 

304 

304 

304 

Mean 

3.15 

2.97 

2.95 

3.14 

27.82 

24.47 

2.38 

2.92 

100.60 

99.50 

102.20 

101.74 

Standard 

deviation 

0.43 

0.51 

0.45 

0.53 

7.96 

10.49 

1.66 

1.64 

1.43 

1.78 

2.17 

2.01 


* Significant at the 0.05 level (two-tailed); ** significant at the 0.01 level (two-tailed). 

WELAIS is writing and English language arts combined index score, MIS is math index score, SCIS is science index score, SSIS is social 
science index score, ELAPA is English language arts weighted points, MATHPA is math weighted points, SCPA is science weighted 
points, SSPA is social science weighted points, ELAG is English language arts growth index, MATHG is math growth index, SCG is sci- 
ence growth index, and SSG is social science growth index. 

Note: Middle school covers grades 6-8. 

Source: Authors’ analysis based on data from the South Carolina Department of Education (2014a). 
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Table B7. Correlations between and means and standard deviations for South Carolina rating index 
measures at the high school level, 2012/13 


1 Measure 

LHSAP 

HSAP 

EOC 

ONTIME 

FIVEYR 

ELAPA 

MATHPA 

SCPA 

SSPA 

GRAD 

GROWTH 1 

LHSAP 

1 











HSAP 

.66** 

1 










EOC 

.57** 

.84** 

1 









ONTIME 

.74** 

.55** 

44** 

1 








FIVEYR 

.73** 

.61** 

.47** 

.91** 

1 







ELAPA 

46** 

.70** 

.58** 

27** 

27** 

1 






MATHPA 

46** 

.80** 

.63** 

.30** 

.33** 

.76** 

1 





SCPA 

29** 

.57** 

.57** 

.13 

.12 

49** 

.57** 

1 




SSPA 

_41** 

.50** 

64** 

.31** 

28** 

32** 

.34** 

.31** 

1 



GRAD 

.39** 

.33** 

22** 

48** 

.61** 

22** 

27** 

.15* 

.13 

1 


GROWTH 

20** 

.11 

.07 

.12 

.01 

-.04 

.00 

-.07 

.00 

-.18* 

1 

N 

209 

209 

209 

209 

209 

198 

198 

198 

198 

198 

202 

Mean 

90.41 

78.98 

67.44 

77.53 

78.82 

17.81 

15.63 

3.54 

1.40 

20.30 

0.10 

Standard 












deviation 

11.24 

11.89 

15.03 

13.48 

13.59 

5.77 

6.93 

1.44 

1.45 

8.09 

0.30 


* Significant at the 0.05 level (two-tailed); ** significant at the 0.01 level (two-tailed). 

LHSAP is longitudinal (spring) High School Assessment Program passage rate, HSAP is first attempt High School Assessment Program 
passage rate, EOC is end-of-course exam overall passage rate, ONTIME is four-year cohort graduation rate, FIVEYR is five-year gradua- 
tion rate, ELAPA is English language arts weighted points, MATHPA is math weighted points, SCPA is science weighted points, SSPA is 
social science weighted points, GRAD is graduation rate weighted points, and GROWTH is change in absolute performance index from 
prior year. 

Note: High school covers grades 9-12. 

Source: Authors' analysis based on data from the South Carolina Department of Education (2014a). 


Figure Bl. Sample South Carolina factor model specifications at the elementary and middle school 
levels: One-factor model 



WELAIS is writing and English language arts combined index score, MIS is math index score, SCIS is science index score, SSIS is social 
science index score, ELAPA is English language arts weighted points, MATHPA is math weighted points, SCPA is science weighted 
points, SSPA is social science weighted points, ELAG is English language arts growth index, MATHG is math growth index, SCG is sci- 
ence growth index, and SSG is social science growth index. 

Note: Elementary school covers grades 3-5, and middle school covers grades 6-8. 

Source: Authors' illustration. 
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Figure B2. Sample South Carolina factor model specifications at the elementary and middle school 
levels: Two-factor model 



WELAIS is writing and English language arts combined index score, MIS is math index score, SCIS is science index score, SSIS is social 
science index score, ELAPA is English language arts weighted points, MATHPA is math weighted points, SCPA is science weighted 
points, SSPA is social science weighted points, ELAG is English language arts growth index, MATHG is math growth index, SCG is sci- 
ence growth index, and SSG is social science growth index. 

Note: Elementary school covers grades 3-5, and middle school covers grades 6-8. 

Source: Authors' illustration. 


Figure B3. Sample South Carolina factor model specifications at the elementary and middle school 
levels: Three-factor model 



ESEA is Elementary and Secondary Education Act, WELAIS is writing and English language arts combined index score, MIS is math 
index score, SCIS is science index score, SSIS is social science index score, ELAPA is English language arts weighted points, MATHPA 
is math weighted points, SCPA is science weighted points, SSPA is social science weighted points, ELAG is English language arts 
growth index, MATHG is math growth index, SCG is science growth index, and SSG is social science growth index. 

Note: Elementary school covers grades 3-5, and middle school covers grades 6-8. 

Source: Authors' illustration. 
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Figure B4. Sample South Carolina factor model specifications at the elementary and middle school 
levels: Bi-factor model 



ESEA is Elementary and Secondary Education Act, WELAIS is writing and English language arts combined index score, MIS is math 
index score, SCIS is science index score, SSIS is social science index score, ELAPA is English language arts weighted points, MATHPA 
is math weighted points, SCPA is science weighted points, SSPA is social science weighted points, ELAG is English language arts 
growth index, MATHG is math growth index, SCG is science growth index, and SSG is social science growth index. 

Note: Elementary school covers grades 3-5, and middle school covers grades 6-8. 

Source: Authors' illustration. 


Figure B5. Sample South Carolina factor model specifications at the high school level: One-factor 
model 1 



LHSAP is longitudinal (spring) High School Assessment Program passage rate, HSAP is first attempt High School Assessment Program 
passage rate, EOC is end-of-course exam overall passage rate, ONTIME is four-year cohort graduation rate, FIVEYR is five-year gradua- 
tion rate, GRAD is graduation rate weighted points, ELAPA is English language arts weighted points, MATHPA is math weighted points, 
SCPA is science weighted points, SSPA is social science weighted points, and GROWTH is change in absolute performance index from 
prior year. 

Note: High school covers grades 9-12. 

Source: Authors' illustration. 
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Figure B6. Sample South Carolina factor model specifications at the high school level: One-factor 
model 2 



GROWTH is change in absolute performance index from prior year, LHSAP is longitudinal (spring) High School Assessment Program 
passage rate, HSAP is first attempt High School Assessment Program passage rate, EOC is end-of-course exam overall passage rate, 
ONTIME is four-year cohort graduation rate, FIVEYR is five-year graduation rate, GRAD is graduation rate weighted points, ELAPA is 
English language arts weighted points, MATHPA is math weighted points, SCPA is science weighted points, and SSPA is social science 
weighted points. 

Note: High school covers grades 9-12. 

Source: Authors' illustration. 


Figure B7. Sample South Carolina factor model specifications at the high school level: Two-factor 
model 



ESEA is Elementary and Secondary Education Act, GROWTH is change in absolute performance index from prior year, LHSAP is longi- 
tudinal (spring) High School Assessment Program passage rate, HSAP is first attempt High School Assessment Program passage rate, 
EOC is end-of-course exam overall passage rate, ONTIME is four-year cohort graduation rate. FIVEYR is five-year graduation rate, GRAD 
is graduation rate weighted points, ELAPA is English language arts weighted points, MATHPA is math weighted points, SCPA is science 
weighted points, and SSPA is social science weighted points. 

Note: High school covers grades 9-12. 

Source: Authors' illustration. 
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Figure B8. Sample South Carolina factor model specifications at the high school level: Bi-factor model 



ESEA is Elementary and Secondary Education Act, LHSAP is longitudinal (spring) High School Assessment Program passage rate, HSAP 
is first attempt High School Assessment Program passage rate, EOC is end-of-course exam overall passage rate, ONTIME is four-year 
cohort graduation rate, FIVEYR is five-year graduation rate, GRAD is graduation rate weighted points, ELAPA is English language arts 
weighted points, MATHPA is math weighted points, SCPA is science weighted points, SSPA is social science weighted points, and 
GROWTH is change in absolute performance index from prior year. 

Note: High school covers grades 9-12. 

Source: Authors' illustration. 


only the unidimensional model and the bi-factor model generated a precision weighted 
factor score that could be used to characterize school performance. 

Within the confirmatory factor analysis framework, two alternative models represent a 
hierarchical factor structure: higher order models (also known as second-order models) and 
bhfactor models. Higher order models have been applied in a variety of substantive areas, 
whereas bi-factor models have been used almost exclusively in personality research. Murray 
and Johnson (2013) found that both higher order and bi-factor models yield relatively 
stable estimates of general construct scores. Higher order models assume that the effects 
of the general construct are mediated by lower order constructs, while bi-factor models 
assume that the associations between the general construct and the measures are direct 
and independent of the specific constructs (Murray & Johnson, 2013). In bi-factor models 
the general construct explains the covariance shared by all the measures. The specific 
constructs account for the covariance independent of the general construct. The general 
construct and the specific constructs are uncorrelated and account for the covariance 
simultaneously and independently for each measure. The idea of bi-factor models is to look 
at how much of the total covariance is explained by the specific constructs compared with 
that explained by the general construct (Deng, Wells, & Hambleton, 2008). 

The weak correlations between the growth performance index and the other two indices 
support the assumption that specific constructs are independent of the general construct. 
Higher order models are more suitable when lower order factors are sufficiently correlated 
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with each other, but bi-factor models are applicable to situations when not all the con- 
structs are correlated. 

Other key differences between competing confirmatory factor analysis models are summa- 
rized in box Bl. 

The loadings and residuals for each factor model were examined for statistical significance 
and reasonability, as well as the reliability of the measures and fit of the estimated model 
using multiple indices: 

• Ratio between the chi-square and degrees of freedom (p( 2 /df; Byrne, 1989; < 3.0 
desired). 

• Akaike information criteria (AIC; Kaplan, 2000: smaller values desired). 

• Bayesian information criterion (BIC; Kaplan, 2000; smaller values desired). 

• Root mean square error of approximation (RMSEA; Browne & Cudeck, 1992; 
< .05 desired). 

• Comparative fit index (CFI; Bentler, 1990; > .95 desired). 

• Tucker-Lewis index (TLI; Bentler & Bonnett, 1980; > .95 desired). 

• Standardized root square mean residual (SRMR; Hu & Bentler, 1998; < .05 desired). 


Box Bl. Key differences between confirmatory factor analysis models 

One-factor model 

• Advantage: A single underlying factor (for example, school quality) accounts for the vari- 
ance in all the individual measures. 

• Disadvantage: The model does not account for relationships between specific factors 
(for example, absolute performance and Elementary and Secondary Education Act 
performance). 

Multiple correlated factors model 

• Advantage: The model explicitly accounts for relationships between specific factors. 

• Disadvantage: The model does not account for the effects of a single underlying factor (for 
example, school quality). 

Higher order factor model 

• Advantage: The model includes the effects of both specific factors (for example, absolute 
performance) and a general factor (for example, school quality). 

• Disadvantages: The higher order factor (for example, school quality) does not have a direct 
relationship with (or effect on) individual measures. In addition, the general factor and 
specific factors are not easily included in a prediction equation. 

Bi-factor model 

• Advantages: The model represents simultaneous effects of both specific factors and a 
general latent factor on the individual measures and provides purer estimates of specific 
factors than can be obtained from a higher order factor model. 

• Disadvantage: In practice, the general factor is uncorrelated with the specific factors, 
which could misrepresent the data. 
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The bi'factor model provided the best fit to the data at all school levels (table B8). 
For example, the bi-factor and the next best fitting model at the elementary level (the 
threeTactor model) differed by AAIC = 69 and ABIC = 59 in favor of the bi-factor model 
(j( 2 (33) = 74.23, RMSEA = .04 [90 percent confidence interval = .03, .06], CFI = .99, 
TLI = .98). To contextualize the differences in BIC values, Raftery (1995) reported that 
information criteria differences of greater than 10 provide evidence for large, practically 
important differences. Murray and Johnson (2013) recommend differences in BIC values 
higher than 10, which is demonstrated with differences ranging from 46 to 95. 

Reliability for the school performance general factor was estimated as: 

c hi ) 2 

Reliability of school performance general factor = — , 

(i=i\) + (i=i°.) 


where X is the standardized factor loading for measure i and §. is the error variance for 
measure i. The standardized estimates and residual variances in tables B9 and B10 were 
used in this equation to estimate reliability (Raykov, 1997). 

Factor scores were estimated for the population of schools with a mean set at 0 and a start- 
dard deviation set at 1 so that school performance general factor scores were on a 2 -score 
metric. The factor score determinacies for the school performance general factor were 
found to be very high (.99 at the elementary and middle school levels and .98 at the high 


Table B8. Comparison of model fit indices by school level, 2012/13 


Model 

Chi- 

square 

Degrees of 
freedom 

p value 

Akaike 

information 

criteria 

Adjusted 

Bayesian 

information 

criteria 

Root mean square error 
of approximation 

90 percent 
confidence interval 

Lower Upper 

bound bound 

Comparative 
fit index 

Tucker- 

Lewis 

index 

Standardized 
root mean 
square 
residual 

Elementary schools (grades 3-5) 

One factor 

487.26 

42 

0.00 

23,801 

23,863 

0.13 

0.12 

0.14 

0.88 

0.82 

0.25 

Two factor 

228.34 

42 

0.00 

23,239 

23,301 

0.08 

0.07 

0.09 

0.95 

0.92 

0.09 

Three factor 

128.90 

41 

0.00 

23,085 

23,149 

0.06 

0.05 

0.07 

0.98 

0.96 

0.05 

Bi -factor 

74.23 

33 

0.00 

23,016 

23,090 

0.04 

0.03 

0.06 

0.99 

0.98 

0.07 

Middle schools (grades 6-8) 

One factor 

253.72 

42 

0.00 

10,509 

10,535 

0.13 

0.11 

0.14 

0.87 

0.79 

0.10 

Two factor 

201.80 

43 

0.00 

10,420 

10,445 

0.11 

0.10 

0.13 

0.90 

0.85 

0.11 

Three factor 

135.53 

42 

0.00 

10,303 

10,330 

0.09 

0.07 

0.10 

0.94 

0.91 

0.13 

Bi -factor 

102.84 

33 

0.00 

10,253 

10,284 

0.08 

0.07 

0.10 

0.96 

0.91 

0.09 

High schools (grades 9-12) 

One factor l a 

144.54 

40 

0.00 

12,419 

12,426 

0.11 

0.09 

0.13 

0.89 

0.85 

0.10 

One factor 2 a 

144.54 

40 

0.00 

12,419 

12,426 

0.11 

0.09 

0.13 

0.89 

0.85 

0.10 

Two factor 

140.27 

38 

0.00 

12,406 

12,413 

0.11 

0.09 

0.13 

0.90 

0.85 

0.10 

Bi-factor 

92.25 

34 

0.00 

12,311 

12,318 

0.09 

0.07 

0.11 

0.94 

0.90 

0.07 


a. The model fit does not change between the one-factor models at the high school level because there is only one measure of growth 
at the high school level. Growth at the high school level is calculated as the difference between the current and prior year absolute 
performance index score. 

Source: Authors' analysis based on data from the South Carolina Department of Education (2014a). 
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Table B9. Standardized factor loadings, residual covariances, 
models at the elementary and middle school levels, 2012/13 

and residual variances for the bi-factor 


Elementary schools (grades 3- 

5) 

Middle schools (grades 6- 

-8) 

Measure 

Standardized 

estimate 

Standard 

error 

p value 

Standardized 

estimate 

Standard 

error 

p value 

Absolute performance by 

WELAIS 

- 0.08 

0.03 

0.02 

0.29 

0.10 

0.01 

MIS 

- 0.03 

0.03 

0.43 

0.33 

0.14 

0.02 

SCIS 

- 0.14 

0.03 

0.00 

0.16 

0.10 

0.11 

SSIS 

- 0.44 

0.04 

0.00 

- 0.09 

0.22 

0.68 

Elementary and Secondary Education Act performance by 

ELAPA 

0.32 

0.05 

0.00 

0.52 

0.08 

0.00 

MATHPA 

0.51 

0.07 

0.00 

0.49 

0.07 

0.00 

SCPA 

0.06 

0.05 

0.21 

0.18 

0.08 

0.03 

SSPA 

0.33 

0.06 

0.00 

0.21 

0.08 

0.01 

Growth performance by 

ELAG 

0.78 

0.05 

0.00 

0.71 

0.08 

0.00 

MATHG 

0.76 

0.04 

0.00 

0.65 

0.08 

0.00 

SCG 

0.56 

0.11 

0.00 

0.39 

0.16 

0.02 

SSG 

0.59 

0.11 

0.00 

0.46 

0.09 

0.00 

School performance by 

WELAIS 

0.96 

0.01 

0.00 

0.93 

0.04 

0.00 

MIS 

0.96 

0.01 

0.00 

0.90 

0.05 

0.00 

SCIS 

0.95 

0.01 

0.00 

0.93 

0.02 

0.00 

SSIS 

0.90 

0.02 

0.00 

0.99 

0.02 

0.00 

ELAPA 

0.53 

0.03 

0.00 

0.62 

0.04 

0.00 

MATHPA 

0.70 

0.02 

0.00 

0.66 

0.03 

0.00 

SCPA 

0.61 

0.03 

0.00 

0.71 

0.05 

0.00 

SSPA 

0.62 

0.03 

0.00 

0.76 

0.04 

0.00 

ELAG 

- 0.18 

0.04 

0.00 

- 0.26 

0.06 

0.00 

MATHG 

- 0.15 

0.04 

0.00 

0.05 

0.06 

0.36 

SCG 

- 0.10 

0.05 

0.04 

- 0.19 

0.10 

0.07 

SSG 

- 0.06 

0.05 

0.22 

0.09 

0.07 

0.18 

Residual covariances 

MATHPA with MIS 

0.67 

0.10 

0.00 

0.65 

0.12 

0.00 

SSPA with SSIS 

a 

a 

a 

b 

b 

b 

ELPA with WELAIS 

0.18 

0.05 

0.00 

0.23 

0.10 

0.02 

SCPA with SCIS 

0.37 

0.04 

0.00 

0.42 

0.07 

0.00 

MATHG with MIS 

0.41 

0.06 

0.00 

0.37 

0.14 

0.01 

MATHG with MATHPA 

0.41 

0.08 

0.00 

0.37 

0.09 

0.00 

SCG with SCIS 

0.30 

0.05 

0.00 

0.44 

0.09 

0.00 

SCG with SCPA 

0.30 

0.04 

0.00 

0.37 

0.08 

0.00 

SSG with SSIS 

a 

a 

a 

b 

b 

b 

SSG with SSPA 

0.43 

0.08 

0.00 

0.35 

0.07 

0.00 

ELAG with ELAPA 

b 

b 

b 

0.31 

0.087 

0.00 


(continued) 
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Table B9. Standardized factor loadings, residual covariances, and residual variances for the bi-factor 
models at the elementary and middle school levels, 2012/13 (continued) 



Elementary schools (grades 3- 

-5) 

Middle schools (grades 6- 

-8) 

Measure 

Standardized 

estimate 

Standard 

error 

p value 

Standardized 

estimate 

Standard 

error 

p value 

Residual variances 

WELAIS 

0.08 

0.01 

0.00 

0.05 

0.02 

0.01 

MIS 

0.07 

0.01 

0.00 

0.09 

0.02 

0.00 

SCIS 

0.08 

0.01 

0.00 

0.12 

0.02 

0.00 

SSIS 

0.00 

a 

a 

0.01 

0.08 

0.90 

ELAPA 

0.62 

0.05 

0.00 

0.35 

0.08 

0.00 

MATHPA 

0.25 

0.07 

0.00 

0.33 

0.06 

0.00 

SCPA 

0.63 

0.03 

0.00 

0.46 

0.06 

0.00 

SSPA 

0.51 

0.06 

0.00 

0.38 

0.05 

0.00 

ELAG 

0.36 

0.07 

0.00 

0.42 

0.10 

0.00 

MATHG 

0.41 

0.06 

0.00 

0.58 

0.10 

0.00 

SCG 

0.67 

0.12 

0.00 

0.81 

0.10 

0.00 

SSG 

0.65 

0.12 

0.00 

0.78 

0.09 

0.00 


WELAIS is Index score for writing and English language arts combined, MIS is math index score, SCIS is science index score, SSIS is 
social science index score, ELAPA is English language arts weighted points, MATHPA is math weighted points, SCPA is science weighted 
points, SSPA is social science weighted points, ELAG is English language arts growth index, MATHG is math growth index, SCG is sci- 
ence growth index, and SSG is social science growth index. 

a. Not estimated because elementary residual variance for SSIS was set to 0. 

b. Removed from the model specification to improve fit because error covariances were nonsignificant. 

Source: Authors' analysis based on data from the South Carolina Department of Education (2014a). 


Table BIO. Standardized factor loadings and residual variances for the bi-factor 

model at the high school level, 2012/13 



Measure 

Standardized estimate 

Standard error 

p value 

Absolute performance by 

LHSAP 

0.46 

0.12 

0.00 

HSAP 

0.12 

0.07 

0.11 

EOC 

-0.01 

0.08 

0.93 

ONTIME 

0.77 

0.07 

0.00 

FIVEYR 

0.84 

0.07 

0.00 

GRAD 

0.56 

0.07 

0.00 

Elementary and Secondary Education Act performance by 

ELAPA 

0.32 

0.07 

0.00 

MATHPA 

0.43 

0.11 

0.00 

SCPA 

0.12 

0.08 

0.15 

SSPA 

-0.29 

0.11 

0.01 

School performance by 

LHSAP 

0.64 

0.06 

0.00 

HSAP 

0.95 

0.02 

0.00 

EOC 

0.88 

0.02 

0.00 

ONTIME 

0.50 

0.09 

0.00 

(continued) 
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Table BIO. Standardized factor loadings and residual variances for the bi-factor 
model at the high school level, 2012/13 (continued) 

Measure 

Standardized estimate 

Standard error 

p value 

FIVEYR 

0.54 

0.09 

0.00 

GRAD 

0.40 

0.09 

0.00 

ELAPA 

0.78 

0.04 

0.00 

MATHPA 

0.85 

0.03 

0.00 

SCPA 

0.68 

0.05 

0.00 

SSPA 

0.64 

0.05 

0.00 

GROWTH 

0.12 

0.11 

0.24 

Residual variances 

LHSAP 

0.38 

0.09 

0.00 

HSAP 

0.09 

0.03 

0.00 

EOC 

0.22 

0.04 

0.00 

ONTIME 

0.16 

0.08 

0.04 

FIVEYR 

0.01 

0.07 

0.84 

GRAD 

0.53 

0.10 

0.00 

ELAPA 

0.30 

0.07 

0.00 

MATHPA 

0.09 

0.08 

0.23 

SCPA 

0.52 

0.06 

0.00 

SSPA 

0.51 

0.10 

0.00 

GROWTH 

0.99 

0.03 

0.00 


LHSAP is longitudinal (spring) High School Assessment Program passage rate, HSAP is first attempt High 
School Assessment Program passage rate, EOC is end-of-course exam overall passage rate, ONTIME is four- 
year cohort graduation rate, FIVEYR is five-year graduation rate, GRAD is graduation rate weighted points, 
ELAPA is English language arts weighted points, MATHPA is math weighted points, SCPA is science weighted 
points, SSPA is social science weighted points, GROWTH is change in absolute performance index from prior 
year. 

Note: High school covers grades 9-12. 

Source: Authors’ analysis based on data from the South Carolina Department of Education (2014a). 


school level), indicating a strong, positive correlation between the estimated score and the 
factor score. In other words, the school performance general factor score estimates meet 
validity requirements for serving as substitutes for the factor itself in subsequent statistical 
analyses (Grice, 2001). 

Correlations between the school performance general factor scores and the existing 
indices were found to be strong for the absolute and ESEA performance indices (approach- 
ing unison at the elementary and middle school levels) and weak or nonsignificant for 
the growth performance index (tables B11-B13). This means that the school performance 
general factor scores maintain the same pattern that is observed in the existing indices 
derived with point systems. 
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Table Bll. Correlations between the school performance general factor and 
existing indices at the elementary school level, 2012/13 

Index 

Absolute 

performance 

index 

Elementary 
and Secondary 
Education Act 
performance 
index 

Growth 

performance 

index 

School 
performance 
general factor 

Absolute performance index 

1 




Elementary and Secondary Education Act 
performance index 

.76** 

1 



Growth performance index 

—.20** 

-.03 

1 


School performance general factor 

.99** 

.76** 

_ 22** 

1 


** Significant at the 0.01 level (two-tailed). 

Source: Authors’ analysis based on data from the South Carolina Department of Education (2014a). 


Table B12. Correlations between the school performance general factor and 
existing indices at the middle school level, 2012/13 


Index 

Absolute 

performance 

index 

Elementary 
and Secondary 
Education Act 
performance 
index 

Growth 

performance 

index 

School 
performance 
general factor 

Absolute performance index 

1 




Elementary and Secondary Education Act 
performance index 

.76** 

1 



Growth performance index 

-.13* 

.06 

1 


School performance general factor 

.98** 

.74** 

-.10 

1 


* Significant at the 0.05 level (two-tailed): ** significant at the 0.01 level (two-tailed). 

Source: Authors’ analysis based on data from the South Carolina Department of Education (2014a). 


Table B13. Correlations between the school performance general factor and 
existing indices at the high school level, 2012/13 


Index 

Absolute 

performance 

index 

Elementary 
and Secondary 
Education Act 
performance 
index 

Growth 

performance 

index 

School 
performance 
general factor 

Absolute performance index 

1 




Elementary and Secondary Education Act 
performance index 

.72** 

1 



Growth performance index 

20** 

-.08 

1 


School performance general factor 

.85** 

.83** 

.12 

1 

** Significant at the 0.01 level (two-tailed). 

Source: Authors’ analysis based on data from the South Carolina Department of Education (2014a). 
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Identifying schools that are beating the odds after school demographic characteristics are 
controlled for 

The school performance general factor score estimated by the bi-factor model was used 
as the outcome measure in a multiple linear regression prediction model that used school 
demographic characteristics as predictors: 

Y. = B 0 + Bj(% White) j + B 1 (%Black) j + B 3 (%Hispamc) ] + 

B 4 (%M ale). + B ^(Poverty). + B 6 (%Disabled) . + e., 

where the dependent variable Y. is the value of the school performance general factor score 
for school j, controlling for the school’s poverty index and the percentage of students who 
are classified as White, Black, Hispanic, male, and disabled. The school-level residual, e., 
served as the measure of a school’s success in increasing school performance. 

The initial elementary school dataset included 649 schools. Two cases were deleted due to 
missing poverty index scores. An analysis of univariate and multivariate outliers resulted 
in the deletion of an additional 28 cases. The final dataset included 619 schools. 

The initial middle school dataset included 304 schools. Five cases were deleted due to 
missing poverty index scores. An analysis of univariate and multivariate outliers resulted 
in the deletion of an additional 13 cases. The final dataset included 286 schools. 

The initial high school dataset included 209 schools. An analysis of univariate and multi- 
variate outliers resulted in the deletion of 8 cases. The final dataset included 201 schools. 

Prior to conducting the analyses, the school-level correlations among the demographic 
variables were examined for multicollinearity. The percentages of White and Black stu- 
dents were found to be negatively correlated above -0.90 at all school levels: -0.92 at the 
elementary school level, -0.94 at the middle school level, and -0.96 at the high school 
level. Variance inflation factor statistics supported the assumption that including both 
variables in the models would present multicollinearity issues; variance inflation factor 
values for White and Black students were above 10 in each school-level model. 

To determine the best set of predictors for use in the models, variable selection procedures 
were used. At the elementary school level the adjusted R 2 , Mallows’s C p statistic, and BIC 
methods converged on the inclusion of the following variables in the model: percentage of 
students who are Black, percentage of students who are Hispanic, poverty index value, and 
percentage of students who are disabled. This combination of variables also presented the 
lowest estimated mean square error of prediction. This prediction model resulted in an R 2 
of .69. The effects of the independent variables on the school performance general factor 
score are summarized in table B14. 

Using the same process, the variables selected for the middle school model were percentage 
of students who are Black, percentage of students who are Hispanic, poverty index value, 
and percentage of students who are disabled. However, the percentage of students who are 
disabled was not found to be a significant predictor ( p = .09) so it was removed from the 
final model. This prediction model resulted in an R 2 of .68 (see table B14). 
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Table B14. Regression results summary by school level, 2012/13 

Variable 

Standardized 

estimate 

t value 

p value 

Elementary schools (grades 3-5): n = 619; R 2 = .69 

Intercept 

0.00 

28.35 

0.00 

Percentage of students who are Black 

-0.31 

-9.81 

0.00 

Percentage of students who are Hispanic 

-0.05 

-2.15 

0.03 

Poverty index value 

-0.56 

-16.98 

0.00 

Percentage of students who are disabled 

-0.09 

-3.86 

0.00 

Middle schools (grades 6-8): n = 286; R 2 = .68 

Intercept 

0.00 

19.23 

0.00 

Percentage of students who are Black 

-0.15 

-3.12 

0.00 

Percentage of students who are Hispanic 

0.08 

2.26 

0.02 

Poverty index value 

-0.71 

-15.19 

0.00 

High schools (grades 9-12): n = 201; R 2 = .70 

Intercept 

0.00 

-1.57 

0.12 

Percentage of students who are White 

0.34 

6.02 

0.00 

Percentage of students who are male 

0.17 

4.12 

0.00 

Poverty index value 

-0.42 

-6.65 

0.00 

Percentage of students who are disabled 

-0.22 

-4.71 

0.00 

Source: Authors' analysis based on data from the South Carolina Department of Education (2014a). 


At the high school level the variable selection process resulted in the use of the following 
variables: percentage of students who are White, percentage of students who are male, 
poverty index value, and percentage of students who are disabled. This prediction model 
resulted in an R 2 of .70 (see table B14). 

The model residuals were used to evaluate which schools’ observed estimates were greater 
than expected (that is, which schools performed better than expected by having a higher 
school performance general factor score than expected). 

Because residuals have a mean of zero, the range of residuals was important for contex- 
tualizing the differences between the observed and predicted factor scores. Additionally, 
the standard error of the residual — a measure of the precision of each residual estimate — 
provides information for identifying which schools could reliably be identified as high per- 
formers given the data. The range of the elementary school residuals was -1.60 to 1.78, 
with a residual standard error of .53. The range of the middle school residuals was -1.88 to 
1.60, with a residual standard error of .54. The range of the high school residuals was -1.89 
to 1.77, with a residual standard error of .49. 

A 95 percent confidence interval was constructed around each school’s residual, using the 
formula: 


95 percent confidence interval = 

School residual ± 1.96 x standard error of the school’s residual. 

This confidence interval allowed for the determination of whether a value of zero was just 
as plausible as the estimated residual. When zero is contained within a school’s 95 percent 
confidence interval, it cannot be reliably determined that the school is high performing. 
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Schools were identified as high performers when the 95 percent confidence interval did 
not include zero. 

Of the 304 elementary schools with a positive residual, 18 had a 95 percent confidence 
interval that did not include zero. Therefore, approximately 3 percent of elementary 
schools in South Carolina can be considered to be beating the odds. The residuals of these 
schools ranged from 1.05 to 1.78. 

Of the 146 middle schools with a positive residual, 6 had a 95 percent confidence interval 
that did not include zero. These schools account for about 2 percent of middle schools in 
South Carolina. The residuals of these schools ranged from 1.03 to 1.60. 

Of the 105 high schools with a positive residual, 6 had a 95 percent confidence interval 
that did not include zero. These schools account for about 3 percent of high schools in 
South Carolina. The residuals of these schools ranged from .96 to 1.77. 

To illustrate, the estimated school performance general factor score for the elementa- 
ry school with the greatest positive residual was .69, but its predicted score, after school 
demographic characteristics were controlled for, was -1.09. This means that the school 
performed about 1.78 standard deviations higher than expected and that 1.78 is its residual 
value. The 95 percent confidence interval for this school is .74 to 2. 82. 4 The 95 percent 
confidence interval does not include zero, which indicates a significant difference between 
the estimated and predicted factor scores. Using this approach, it can be estimated that 
any elementary school with a positive residual less than 1.039 and a standard error of .53 
would no longer be considered as a high performing school because its 95 percent confb 
dence interval (calculated using residual ± 1.96 x .53) would include zero. 

Identifying profiles of demographically similar schools 

The next stage of the analysis featured a latent profile analysis of school demographic 
characteristics. This clustering of schools provides a “nearest neighbor” comparison, allow- 
ing the South Carolina Department of Education to compare the residual scores from the 
regression analyses with those that are most similar based on demographic characteristics. 
The demographic variables used in this analysis included the school’s poverty index and 
the percentage of students who are Black, Hispanic, male, and disabled. Due to the high 
correlations between the percentages of students who are Black and White, the percentage 
of students who are White was not used in the analyses. 

Latent profile analysis is typically used to classify individuals into groups based on their 
responses on a single exam or based on their scores on multiple exams. And though it 
has been used predominantly for diagnostic purposes in psychology, it is an emerging 
descriptive classification technique in education (Koon, Petscher, & Foorman, 2014; Logan 
& Petscher, 2010). While latent profile analysis is descriptive, its utility lies in its ability 
to empirically categorize participants (in this case, schools) into similar groups based on 
variables of interest (in this case, demographic characteristics). For example, if the latent 
profile analysis was used to identify two profiles (or groups) of schools, there would most 
likely be one group of schools with high percentages of students at high risk of reading 
difficulty and one group with low percentages. The schools with high percentages would 
be considered high risk, and those with low percentages would be considered low risk. 
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Then, within a profile, the variation in individual school outcomes can be described (for 
example, school residual scores). 

The basic representation of a multivariate latent profile analysis model (Pastor, Barron, 
Miller, & Davis, 2007) is: 


/(#) = 


where y i represents the multivariate distribution of cluster measures (school demographics) 
for school i (with the number of clusters represented by k), 0 represents the unique set of 
model parameters to be estimated within each cluster, and n k is the weight given to each 
cluster. The weights are constrained to be non-negative and must sum to 1. Each cluster 
distribution is defined by (the mean vector) and £ ( , (the covariance matrix). 

Multiple indices were used to determine which number of profiles was the most appropriate 
for the data (table B15). The indices include Akaike information criteria (Kaplan, 2000), 
Bayesian information criteria (Kaplan, 2000), entropy (Ramaswamy, DeSarbo, Reibstein, 
& Robinson, 1993), and two tests reported in the Mplus program (Muthen & Muthen, 
2012) — the Lo'MendelhRubin likelihood ratio test (Lo, Mendell, & Rubin, 2001) and a 
parametric bootstrapped likelihood ratio test (McLachlan & Peel, 2000). 


A five-class model was selected as the best fit to the data at the elementary school level. 
Moving to six classes resulted in a nonsignificant p-value for the LoMendelhRubin 


Table B15. Summary of latent profile analysis model fit indices by school level, 2012/13 

Number of 
classes 

Akaike 

information 

criteria 

Bayesian 

information 

criteria 

Adjusted 

Bayesian 

information 

criteria 

Entropy 

Lo-Mendell-Rubin adjusted 
likelihood ratio test 

Value p value 

Bootstrapped 
likelihood ratio 
test 

(p value) 

Elementary schools (grades 3-5) 

1 

21,875.07 

21,919.35 

21,887.60 

na 

na 

na 

na 

2 

21,468.79 

21,539.64 

21,488.84 

0.85 

407.71 

0.00 

0.00 

3 

21,100.94 

21,198.35 

21,128.51 

0.90 

370.25 

0.00 

0.00 

4 

20,834.53 

20,958.52 

20,869.62 

0.89 

271.37 

0.00 

0.00 

5 

20,761.52 

20,912.08 

20,804.13 

0.85 

82.86 

0.00 

0.00 

6 

20,693.58 

20,870.71 

20,743.71 

0.88 

77.92 

0.14 

0.00 

Middle schools (grades 6-8) 

1 

9,976.48 

10,013.04 

9,981.33 

na 

na 

na 

na 

2 

9,766.34 

9,824.84 

9,774.10 

0.87 

215.78 

0.00 

0.00 

3 

9,602.20 

9,682.63 

9,612.87 

0.90 

171.10 

0.00 

0.00 

4 

9,531.37 

9,633.74 

9,544.95 

0.89 

80.46 

0.01 

0.00 

5 

9,478.49 

9,602.79 

9,494.98 

0.86 

63.02 

0.11 

0.00 

High schools (grades 9-12) 

1 

6,870.77 

6,903.80 

6,872.12 

na 

na 

na 

na 

2 

6,699.73 

6,752.58 

6,701.89 

0.84 

177.46 

0.00 

0.00 

3 

6,619.39 

6,692.06 

6,622.36 

0.88 

89.53 

0.03 

0.00 

4 

6,567.18 

6,659.67 

6,570.96 

0.89 

62.25 

0.16 

0.00 

na is not applicable. 







Source: Authors' analysis based on data from the South Carolina Department of Education (2014a). 
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likelihood ratio test. In addition to fitting the data well, the five-class model resulted in a 
solution that lent itself well to interpretation. Using the same criteria for selection, a four- 
class model was selected for the middle school level and a three-class model was selected 
for the high school level. 
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Notes 


1. The South Carolina Department of Education uses the term “disabled students” to 
refer to students in special education and “limited English proficient students” to refer 
to English learner students. 

2. The weighted measures for the absolute performance index differ considerably from 
the weighted measures used in the ESEA performance model. The absolute perfor- 
mance index is a weighted average of the total number of students passing a subject 
area test, while the ESEA performance index is based on the total number of sub- 
groups that meet a predetermined proficiency target, with each subgroup worth up to 
one point. 

3. The South Carolina Department of Education uses the term “disabled students” to 
refer to students in special education and “limited English proficient students” to refer 
to English learner students. 

4. The 95 percent confidence interval was created using the following calculations: 1.78 - 
(1.96 x .53) = .74 and 1.78 + (1.96 x .53) = 2.82. 
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The Regional Educational Laboratory Program produces 7 types of reports 



Making Connections 

Studies of correlational relationships 


Making an Impact 

Studies of cause and effect 


What’s Happening 

Descriptions of policies, programs, implementation status, or data trends 


What’s Known 

Summaries of previous research 


Stated Briefly 

Summaries of research findings for specific audiences 


Applied Research Methods 

Research methods for educational settings 


Tools 

Help for planning, gathering, analyzing, or reporting data or research 




